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ABSTRACT. 

Since  2003,  the  Portuguese  Ministry  of  Defense  (MoD)  has  been  using  the  MACBETH  multicriteria 
approach  in  several  evaluation  processes  for  the  acquisition  of  military  equipment.  This  paper  describes 
the  first  application  case:  the  acquisition  of  8x8  Armored  Wheeled  Vehicles  to  equip  the  Army  and  the 
Navy  Marines,  which  contract  was  recently  signed.  This  Tender  was  conducted  by  the  National 
Armaments  Directorate  in  cooperation  with  the  Army  and  the  Navy.  The  process  last  almost  two  years  and 
involved:  on  a  first  stage,  the  revision  and  harmonization  of  the  Sendees’  requirements,  and  the  issuing  of 
the  Tender  Program  and  of  the  Technical  Specifications;  the  process  proceeded  with  the  initial  proposals  ’ 
analysis,  the  performance  of  tests  in  Portugal  and  abroad,  and  the  negotiation  meetings;  the  final  stage  of 
the  Tender  comprehended  the  admission  and  evaluation  of  the  Best  and  Final  Offers,  and  finally  the 
award  of  the  contract  to  the  globally  most  attractive  proposal.  Tender’s  Evaluation  Regulation  was  based 
on  the  MACBETH  (Measuring  Attractiveness  by  a  Categorical  Based  Evaluation  Technique) 
methodology,  previously  adopted  on  large  national  and  international  tender  procedures,  that  proved  to  be 
a  flexible,  consistent  and  robust  decision  support  tool. 


1.  INTRODUCTION 

The  8x8  Armored  Wheeled  Vehicles  (AWV)  acquisition  program,  for  the  Portuguese  Army  and  Navy, 
whose  tender  phase  was  concluded  on  2005,  was  a  paradigmatic  example  of  a  successfully  concluded  joint 
military  equipment  acquisition  program  conducted  by  the  Portuguese  Ministry  of  Defense  (MoD). 
Considering  the  high  complexity  of  the  process,  one  of  key  success  factors  was  the  introduction  by  the 
MoD  of  new  methodologies  to  support  the  decision-making  process  of  the  Tender’s  Commission. 

The  genesis  of  the  program  is  related  with  a  capability  gap  analysis  study  performed  considering  the 
international  commitments  assumed  by  Portugal  and  the  scenarios  where  the  Portuguese  Armed  Forces 
might  be  operating.  Based  on  this  analysis  it  was  set  as  a  priority  to  equip  the  Army  and  the  Navy  Marines 
with  AWV.  Thus,  by  the  end  of  2002  the  preliminary  works  for  the  launching  of  a  competitive  bidding  for 
the  acquisition  of  such  vehicles  began.  The  initial  works  regarding  the  definition  of  the  operational 
requirements  was  carried  by  the  Services’  Staffs. 

In  order  to  ensure  maximum  competitiveness  on  the  AWV  acquisition  program,  a  “public  competitive 
bidding  with  selection  of  proposals  for  negotiation”  was  launched.  This  decision  was  taken  by  the  Minister 
of  Defense  on  July  of  2003,  marking  the  official  start  of  the  program.  The  conduction  of  the  AWV 
acquisition  program  was  assigned  to  the  MoD’s  National  Armaments  Directorate  (DGAED  -  see 
http://www.mdn.gov.pt/mdn/pt/mdn/organograma/dgaed/).  This  is  the  entity  responsible,  among  others 

1  Paper  presented  at  the  SAS-080  Specialists  Meeting  on  "Decision  Support  Methodologies  for  Acquisition  of  Military 
Equipment",  Brussels,  Belgium,  October  22-23,  2009. 
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activities,  for  the  supervision,  coordination,  control  and  execution  of  the  Armed  Forces  equipment 
programs. 

This  paper  presents  a  general  overview  of  this  acquisition  process.  Section  2  describes  the  phases  of  the 
bidding  procedure.  In  Section  3,  special  emphasis  will  be  given  to  the  evaluation  methodology,  in 
particular  regarding  the  application  of  the  MACBETH  approach  (Measuring  Attractiveness  by  the 
Categorical  Based  Evaluation  Technique)  ( cf.  Bana  e  Costa  et  al.  2003;  Bana  e  Costa  and  Vansnick  1994). 
This  was  a  socio-technical  process  that  evolved  throughout  a  sequence  of  decision  conferences  (Phillips 
and  Bana  e  Costa,  2007)  facilitated  by  the  first  author  of  this  paper,  under  a  contract  signed  between  the 
MoD  and  1ST  (“Instituto  Superior  Tecnico”  -  the  engineering  institute  of  the  Technical  University  of 
Lisbon),  with  the  second  author  acting  as  internal  analyst  and  process  coordinator. 


2.  PHASES  OF  THE  COMPETITIVE  BIDDING  PROCEDURE 

The  AWV  acquisition  program  involved  the  following  phases,  which  are  briefly  presented  ahead:  a 
preparatory  phase,  the  tender  announcement,  the  proposal  preparation  and  clarifications  phase,  the 
proposal  selection  phase,  the  negotiation  phase,  the  contract  awarding  and  the  contract  signature.  The 
program  timeline  is  synthesized  in  Figure  1 . 


Figure  1  -  Portuguese  Army  and  Navy  8x8  AWV  acquisition  program  timeline 


Preparatory  phase 

The  works  regarding  the  8x8  Armored  Wheeled  Vehicles  Acquisition  program,  for  the  Army  and  the 
Navy,  led  by  the  DGAED,  were  initiated  in  July  of  2003,  with  the  review  of  the  tender  program  and  of  the 
technical  specifications  proposed  by  the  Services.  The  work  was  developed,  since  the  beginning,  by  one 
joint  Technical  Team,  that  included  representatives  from  the  MoD,  the  Army,  and  the  Navy,  as  well  as 
civilian  advisors  and  was  headed  by  the  second  author  of  this  paper.  The  activities  of  this  team  covered  the 
technical,  legal  and  financial  issues.  The  works  regarding  Offsets  were  carried  out  by  a  specialized 
“Offsets  Commission”,  which  worked  in  parallel  with  the  technical,  legal  and  financial  teams.  At  the  same 
time,  an  agreement  was  signed  with  1ST,  under  which  a  MACBETH  decision-analysis  process 
consultation  started  in  August  2003.  The  explicit  choice  of  the  MACBETH  evaluation  methodology  by  the 
MoD  was  justified  by  the  success  of  its  application  in  bid  evaluation  on  large  national  and  international 
public  call  for  tenders  ( cf.  Bana  e  Costa  et  al.,  2002).  The  methodology  was  presented  and  discussed  at  an 
initial  meeting  at  the  MoD,  with  emphasis  on  its  alignment  with  the  phases  of  the  bidding  procedure. 

The  main  goal  in  the  preparatory  phase  was  the  requirements  validation,  considering  two  basic  principles: 

•  based  on  the  knowledge  about  the  market,  the  mandatory  requirements  could  not  restrain  the 
number  of  solutions  to  one; 
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•  the  number  of  mandatory  requirements  should  be  minimal.  The  proposal  selection  based  on 
desirable  characteristics  and  on  the  key  proposal  features  should  be  addressed  by  the  evaluation 
methodology. 

The  Tender  Commission,  headed  by  the  National  Armaments  Director,  was  empowered  by  the  Minister  of 
Defense  and  the  approval  of  the  final  tender  program  documentation  took  place  at  the  end  of  August.  The 
tender  was  publicly  announced  in  August  14,  2003.  The  tender  documentation  was  made  available  to  the 
potential  bidders  in  September  1,  2003. 

Proposal  preparation  and  clarifications  phase 

September  1,  2003  marked  the  beginning  of  the  first  phase  of  the  tender  process,  when  the  tender 
documentation  was  made  available  for  consultation  and  acquisition  by  any  potential  tenderer.  During  this 
period  the  documentation  was  consulted  by  6  companies  and  acquired  by  4.  In  the  subsequent  months 
potential  tenderers  were  provided  with  the  requested  clarifications.  Meanwhile,  the  Proposals  Evaluation 
Regulation  was  concluded  and  delivered  to  the  potential  tenderers.  Section  3  is  dedicated  to  an  in-depth 
discussion  of  the  aspects  regarding  the  adopted  evaluation  methodology.  Nevertheless,  in  summary  it  can 
be  said  that,  during  this  phase,  the  works  of  the  Technical  Team  consisted  on  the  structuring  activities 
leading  to  the  identification  of  screening  (rejection)  criteria  and  evaluation  criteria,  the  structuring  of  the 
latter  in  a  value  tree  with  several  levels  of  specification,  and  their  operationalization  throughout  the 
definition  of  reference  performance  levels  (and  value  functions  in  a  few  cases)  validated  by  the  Tender 
Commission  that,  finally,  could  assign  weights  to  the  evaluation  criteria. 

Proposal  Selection  phase 

The  Proposal  Selection  phase  was  initiated  with  the  opening  of  the  Proposals,  which  took  place  on  January 
27,  2004.  The  end  of  subsequent  analysis  work  happened  when  the  Tender  Commission  issued  the 
Preliminary  Report  on  the  Proposal  Selection  for  the  Negotiation  Phase  to  the  Minister  of  the  Defense. 
This  report  was  homologated  on  May  6,  2004.  After  a  period  of  tenderers’  hearings,  the  Final  Report  of 
the  Proposal  Selection  Phase  was  homologated.  The  report  admitted  to  the  Negotiation  phase  3  tenderers: 
Austrian  Steyr-Daimler-Puch  Spezialfahrzeug  AG  &  Co  KG,  with  vehicle  PANDUR  II;  Swiss  Mowag 
Motorwagen  Fabrik  AG,  with  the  vehicle  Piranha  III  C;  and  Finish  Patria  Vehicles  OY,  with  vehicle 
AWV. 

During  the  Selection  Phase  it  was  also  prepared  the  Tests  Plan  for  testing  the  vehicles  in  Portugal.  This 
Plan  defined  the  scenarios  and  the  procedures  for  testing  and  verifying  the  vehicles  and  systems  (about  60 
different  specific  tests)  and  provided  the  test  sheets  to  use  on  the  tests.  Tests  included,  for  instance,  the 
verification  of  the  dimensional  and  functional  characteristics  of  the  vehicles,  of  ergonomic  factors 
affecting  crew  and  carried  personnel,  as  well  as  tests  on  operational  environment,  both  for  terrestrial  and 
amphibious  configurations. 

Negotiation  phase 

This  phase  was  initiated  with  the  submission  of  the  proposed  vehicles  for  performing  the  tests,  which  were 
designed  to  gather  relevant  information,  namely  regarding  the  mandatory  requirements  of  the  Technical 
Specifications  and  the  technical  requirements  of  the  Evaluation  Methodology.  These  tests  were  performed 
to  the  basic  terrestrial  and  amphibious  variants  throughout  a  period  of  three  weeks  on  July  and  August  of 
2004,  in  several  Army,  Navy  and  Air  Force  premises,  involving  not  only  military  but  also  civilian  experts 
on  different  fields  of  expertise.  The  content  of  each  specific  test  was  only  disclosed  to  the  tenderers  at  the 
beginning  of  the  test,  in  a  briefing  presenting  all  the  details  regarding  its  execution,  the  involved  means, 
the  place  where  it  was  to  be  carried  out,  as  well  as  the  safety  aspects  to  observe  (see  Figure  2). 
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Figure  2  -  Tests  performed  in  Portugal 

On  visits  abroad  it  was  possible  to  get  acquainted  with  the  capabilities  of  the  potential  subcontractors  and 
with  the  characteristics  of  the  main  systems  to  install  in  the  AWV,  particularly  the  weapon  systems. 

The  information  gathered  during  the  period  of  tests  and  with  the  visits  and  tests  abroad,  became  decisive 
for  the  conduction  of  the  negotiation  meetings,  which  were  carried  out  between  September  and  October  of 
2004,  and  also  for  the  bid  evaluation  using  the  evaluation  methodology.  The  negotiations  were  done  in 
several  meetings  with  each  of  the  tenderers,  where  the  technical  solutions  and  alternative  proposals  were 
discussed  and  analyzed  in  order  to  improve  the  global  quality  of  the  proposal.  Resulting  from  these 
meetings  some  adjustments  to  the  initial  version  of  the  Tender  Specs  (e.g.,  vehicle  variants,  quantities,  and 
configuration  requirements)  were  communicated  to  the  tenderers,  so  that  they  could  prepare  their  final 
offers. 

Besides  the  technical  negotiations,  other  negotiations  meetings  were  done  with  the  three  tenderers  for 
discussion  of  the  Legal  issues  (aiming  the  agreement,  in  principle,  of  the  wording  of  Contract  minutes) 
and  of  Offsets  (aiming  the  discussion  of  the  Offsets  programs). 

In  the  beginning  of  November  the  tenderers  received  the  directives  for  BAFO  (Best  And  Final  Offer),  i.e., 
the  procedural  and  formal  rules  regarding  the  elaboration,  presentation  and  opening  of  the  final  version  of 
the  proposals.  The  public  opening  of  the  BAFO  occurred  in  the  November  11,  2004  on  the  day  after  the 
date  set  for  the  delivery  of  the  proposals.  Due  to  non-compliance  with  formal  tender  requirements  the 
BAFO  of  Patria  Vehicles  OY  was  not  accepted. 

The  proposals  of  the  two  remaining  tenderers  (hereafter  designated  by  “S”  of  Steyr  and  “M”  of  Mowag) 
were  analyzed  by  the  Technical  Team  and  compared  with  the  reference  performance  levels  defined  on 
each  criterion.  This  work  informed  the  evaluation  of  the  two  proposals  by  the  Tender  Commission, 
according  to  the  tender  regulations.  The  evaluation  activities  were  developed  in  decision  conferences  in 
which  the  M-MACBETH  decision  support  system  (Bana  Consulting,  2005)  facilitated  the  comparison  and 
scoring  of  the  proposals  in  the  evaluation  criteria.  Subsequently,  the  Tender  Commission  delivered  the 
Preliminary  Report  of  the  Negotiation  Phase,  which  was  approved  by  the  Minister  of  Defense  in 
December  1st,  2004.  PANDUR  II  was  the  selected  vehicle  (see  http://www.army- 
guide.com/eng/productl  12.html). 
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Contract  awarding 

After  hearing  the  two  tenderers,  the  Final  Report  of  the  Negotiation  Phase  was  issued  by  the  Tender 
Commission  and  submitted  to  the  approval  of  the  MoD.  The  Final  Report  homologation  occurred  on 
December  6th,  and  with  it  the  Contract  was  awarded  to  the  S  tenderer.  After  the  conclusion  of  the  wording 
review,  two  contracts  were  signed  on  February  15,  2005.  The  Tender  Contract  regarded  the  acquisition  of 
240  terrestrial  AWV  for  the  Army,  and  20  amphibious  AWV,  for  the  Navy.  Table  1  presents  some  data 
about  the  variants  that  were  contracted.  A  second  contract  was  signed  addressing  the  supply  of  spare  parts 
for  scheduled  and  unscheduled  maintenance,  regarding  a  10  years  period  in  the  life  cycle  of  each  AWV. 


Table  1  -  Contracted  AWV  Variants 


Variant 

Terrestrial  AWV 
(Army) 

Amphibious  AWV 
(Navy) 

Infantry  Carrier  Vehicle 

105 

13 

Infantry  Carrier  with  RWS  Vehicle 

7 

- 

Command  Post  Vehicle 

16 

3 

Mortar  Carrier 

31 

2 

Recovery  Vehicle 

7 

- 

Medical  Evacuation  Vehicle 

10 

- 

Anti-tank  Guided  Missile  Vehicle 

15 

2 

Mobile  30  mm  System  Vehicle 

30 

Reconnaissance  Vehicle 

4 

- 

Engineer  Squad  Vehicle 

9 

- 

Armored  Communications  Vehicle 

6 

- 

Mobile  105mm  Cannon  Vehicle 

33  (optional) 

- 

Total  AWV 

240 

+  33  (optional) 

20 

3.  THE  EVALUATION  PROCESS 

3.1  Overview  of  the  Evaluation  Methodology 

The  evaluation  criteria  and  of  the  respective  weights  should  be  announced  to  the  potential  tenderers.  This 
was  a  legal  obligation  according  to  Portuguese  law  and  it  is  now  clear  in  the  recent  Directive  2009/81/EC 
of  the  European  Parliament  and  of  the  Council  of  13  July  2009  (on  the  coordination  of  procedures  for  the 
award  of  certain  works  contracts,  supply  contracts  and  service  contracts  by  contracting  authorities  or 
entities  in  the  fields  of  defense  and  security):  “it  is  therefore  the  responsibility  of  the  contracting 
authorities/entities  to  indicate  the  criteria  for  the  award  of  the  contract  and  the  relative  weighting  given  to 
each  of  those  criteria,  in  sufficient  time  for  tenderers  to  be  aware  of  them  when  preparing  their  tenders” 
(page  L  216/85  §70). 

It  is  well-known  in  the  Decision  Analysis  literature  ( cf  Belton  and  Stewart,  2002)  that  weights  have  no 
operational  meaning  without  reference  to  a  specific  multicriteria  aggregation  procedure.  The  evaluation  of 
the  proposals  of  the  competitive  bidding  for  acquisition  of  the  8x8  AWV  was  done  by  developing,  with 
MACBETH,  a  hierarchical  additive  aggregation  model:  value  scores  are  (directly  or  indirectly)  assigned  to 
each  proposal  and  multiplied  by  the  respective  weights  previously  assigned  to  the  criteria  and  those 
products  are  summed  across  all  of  the  criteria,  at  the  different  levels  of  the  value  tree,  to  determine  the 
overall  value  score  of  each  proposal.  MACBETH  requires  only  qualitative  (non-numerical)  judgments  of 
difference  in  attractiveness  (value)  in  order  to  help  weighting  the  criteria  and  scoring  the  proposals  (Bana 
e  Costa  and  Chagas,  2004).  The  use  of  a  non-numerical  pairwise  comparison  value-elicitation  procedure 
distinguishes  MACBETH  from  the  numerical-based  multicriteria  decision  analysis  procedures  for  bid 
evaluation  surveyed  in  Bana  e  Costa  et  al.  (2008). 


RTO-MP-SAS-080 


1  -5 


Procurement  Decision  Support  for  the  Portuguese  MoD: 
The  MACBETH  Approach  and  the  Acquisition  of  8x8  AWV 


Methodologically,  the  MACBETH  model  building  process,  used  in  the  context  of  the  acquisition  program 
of  8x8  AWV  for  the  Portuguese  Army  and  the  Navy,  can  be  described  as  a  package  of  activities  of 
Structuring,  Weighting,  Scoring  and  Recommending,  developed  during  working  sessions  of  the  Technical 
Team  and  decision  conferences  with  the  Tender  Commission,  as  presented  with  more  detail  hereafter. 

3.2  Definition  of  the  Evaluation  Regulation 

The  Evaluation  Regulation  should  include  the  screening  and  evaluation  criteria,  the  reference  performance 
levels  on  the  evaluation  criteria  and  their  relative  weights.  A  few  value  functions  were  also  defined. 

First  structuring  activity:  defining  screening  criteria  and  evaluation  criteria 

For  the  8x8  AWV  Acquisition  program  it  was  assumed  that  all  the  243  requirements  defined  on  the 
Technical  specifications  of  the  Tender  were  mandatory.  In  this  way,  the  screening  or  rejection  criteria 
coincided  with  the  mandatory  requirements  (also  referred  as  the  essential  requirements  of  the  12  AWV 
variants. 

As  remarked  by  Bana  e  Costa  et  al.  (2008)  the  evaluation  criteria  should  be  carefully  selected;  no 
additional  criteria  can  legally  be  added  after  the  bids  are  submitted.  Moreover,  since  proposals  will  be 
compared  and  scored  in  terms  of  their  relative  attractiveness  with  respect  to  each  criterion  individually, 
each  criterion  must  be  an  independent  evaluation  axis.  This  is  the  reason  why  several  characteristics  are 
often  clustered  into  a  single  criterion  rather  than  analyzed  as  independent  criteria.  The  set  of  criteria 
should  be  consensual,  and  therefore  exhaustive,  but  also  concise  and  non-redundant.  In  addition,  each 
criterion  should  be  specific  and  understandable  and  operational  for  the  appraisal  of  bid  performances. 

The  definition  of  the  evaluation  criteria  started  by  identifying  the  requirements  defined  in  the  Technical 
Specifications  that  are  deemed  as  deserving  a  specific  evaluation,  and  other  characteristics  that,  despite  not 
being  of  mandatory  inclusion  in  the  offers,  would  introduce  some  degree  of  differentiation  in  the  quality  of 
the  proposals.  The  identification  of  the  criteria  and  the  respective  logical  structuring  in  a  value  tree  format 
(see  Figure  3)  was  validated  in  a  decision  conference  with  a  Tender  Commission,  supported  by  previous 
work  by  the  Technical  Team. 

The  value  tree  integrates  three  levels  of  evaluation  criteria  -  criteria,  sub-criteria  and  sub-sub-criteria  - 
that  were  weighted  and  on  which  proposals  should  be  scored.  Four  top-level  criteria  appear  immediately 
after  the  root  node:  ‘Aspects  regarding  Costs’,  ‘Aspects  regarding  Offsets',  ‘Aspects  regarding  Delivery 
Schedule'  and  ‘Aspects  regarding  Technical  Requirements'.  The  first  and  third  criteria  have  no  children 
criteria;  but  there  are  13  sub-criteria  specifying  the  Delivery  Schedule  criterion,  each  one  corresponding 
with  a  specific  AWV  mission  objective.  The  structure  of  the  tree  under  the  Requirements  criterion  node  is 
more  complex,  with  three  sub-criteria  of  ‘Operational’,  ‘Technical  (generic)’  and  ‘Logistical’ 
Requirements,  each  one  in  turn  parent  of  several  sub-sub-criteria.  Each  of  these  sub-sub-criterion  is  a 
group  of  several  more  elementary  requirements.  These  are  intertwined  characteristics  that  cannot  be 
defined  as  isolated  evaluation  criteria.  For  example,  the  AWV  value  tree  in  zoomed  in  Figure  3  to  show 
the  7  sub -sub -criteria  that  specify  the  ‘Operational  Requirements’  and  the  4  characteristics  of  the  ‘AWV 
Combat  System’  defined  at  two-lower  levels.  It  is  important  to  emphasize  that  the  characteristics  are  not 
independent  evaluation  criteria  and  therefore  they  cannot  be  weighted  nor  can  the  proposals  be  scored 
separately  on  each  of  them. 
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|-|Tender  for  Acquisition  of  8x8  A\W  | 
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Figure  3  -  Partial  view  of  the  AWV  value  tree. 

Second  structuring  activity:  Defining  reference  performance  levels 

Two  reference  performance  levels  of  intrinsic  value  (‘were  defined  on  each  evaluation  criterion.  They 
convey  and  make  operational  the  concept  of  neutral  proposal  (i.e.,  neither  attractive  nor  repulsive)  and  of 
good  proposal.  The  identification  of  the  levels  Good  and  Neutral: 

•  contributed  to  increase  the  intelligibility  of  the  criteria; 

•  made  possible  to  define  the  notion  of  intrinsic  attractiveness  of  each  proposal  (in  order  to  be 
categorized  as  very  positive,  positive  or  negative);  and 

•  allowed  to  use  a  criteria  weighting  procedure,  simultaneously  adjusted  to  the  specific  characteristics 
of  the  competitive  bidding  and  valid  regarding  the  theoretical  framework  of  additive  value 
aggregation. 

The  reference  levels  were  defined  from  bottom  to  up  in  the  value  tree.  For  example,  Table  2  shows  the 
‘Good’  and  ‘Neutral’  performances  for  each  of  the  four  characteristics  of  the  ‘AWV  Combat  Systems’. 
Moving  up  one  level  in  the  tree,  a  Good  (or  Neutral)  ‘AWV  Combat  Systems’  would  be  one  with  good  (or 
neutral)  performances  in  all  the  4  characteristics  (or  another  profile  as  attractive  as  this  one). 
Multidimensional  reference  levels  were  defined  in  a  similar  way  for  all  other  sub-sub-criteria  that  are 
defined  upon  characteristics.  And  so  on  along  the  tree.  Figure  4  shows  the  reference  levels  for  the  4 
evaluation  criteria. 
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Table  2  -  Example  of  the  reference  performance  levels  for  the  sub-sub-criterion  ‘AWV  Combat  Systems’ 


Figure  4  -  Performance  reference  at  the  top-level  of  the  value  tree. 


Of  course,  reference  levels  were  directly  specified  for  those  sub-sub-criteria  or  sub-criteria  or  criteria  that 
have  no  children.  This  is  the  less  complex  situation.  On  the  other  hand,  when  the  degree  of  complexity  is 
very  high  due  to  the  presence  of  a  significant  number  of  intertwined  characteristics,  the  definition  of  the 
reference  levels  can  be  facilitated  by  adopting  the  “determinants  technique”  proposed  by  Bana  e  Costa  et 
al.  (2002). 

On  the  defined  Evaluation  Methodology,  in  each  evaluation  criterion,  the  Neutral  level  corresponded  to  a 
0  (zero)  score  and  the  Good  level  to  a  100  (one  hundred)  score. 

The  weighting  activity 

The  process  of  weighting  the  evaluation  criteria  was  developed  in  a  hierarchical  way,  in  decision 
conferences  with  the  support  of  the  M-MACBETH  software.  This  enabled  to  construct  an  additive  value 
model  of  type  (1)  to  aggregate  the  scores  Vj(b)  of  each  bid  b  in  the  77=4  criteria  (j  -  1,  . . .,  n)  and  calculate 
its  overall  score  V(b). 
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n  n 

V(b )  =  y'i  k ,  .v  ■  (b)  with  =1  and  kj>  0  and 

7=1  7=1 


(vj(goodj)  =  100 
I  Vj  (neutral j )  =  0 


(1) 


The  reference  performance  levels  in  each  criterion  j  are  designated  by  goodj  and  neutral ).  They  allow  to 
determine  the  scaling  factors  kj  -  commonly  known  as  “weighting  coefficients”  or  relative  “weights”  - 
that  in  turn  allow  value  units  on  different  criteria  to  be  transformed  into  overall  value  units,  through  some 
form  of  operationaling  the  notion  of  value  trade-off:  how  much  the  Tender  Commission  considers 
necessary  to  improve  the  performance  in  one  criterion  to  compensate  a  decrease  of  performance  on 
another  criterion. 


Let  us  detail  the  MACBETH  weighting  process  that  took  place  at  the  top-level  of  the  four  valuation 
criteria.  Firstly,  to  order  the  criteria  weights,  it  was  sufficient  to  ask  the  Tender  Commission:  “Consider  a 
hypothetical  proposal  (N),  with  neutral  performances  in  all  criteria.  If  it  were  possible  to  improve  the 
performance  of  N  from  neutral  to  good  in  only  one  criterion,  maintaining  all  of  the  others  at  their  neutral 
level,  for  which  criterion  would  this  swing  be  most  attractive?  And  the  next  most  attractive?”  Repeating 
this  question  until  no  criteria  are  left,  led  directly  to  the  order  of  importance  of  the  swings  and 
consequently,  by  model  (1),  to  the  order  of  the  criteria  weights.  The  evaluation  criteria  were  therefore 
presented  to  the  potential  tenderers  already  in  decreasing  order  of  their  relative  weights: 

1st  Aspects  regarding  Costs; 

2nd  Aspects  regarding  Technical  Requirements; 

3rd  Aspects  regarding  Offsets; 

4th  Aspects  regarding  Delivery  Schedule. 


Next,  to  estimate  numerical  values  for  the  criteria  weights  with  MACBETH,  the  Tender  Commission 
started  by  making  qualitative  judgments  about  the  relative  importance  of  each  one  of  the  four  swings  from 
neutral  to  good.  The  judgements  are  expressed  by  choosing  one  of  the  MACBETH  semantic  categories 
(“very  weak”,  “weak”,  “moderate”,  “strong”,  “very  strong”,  or  “extreme”  difference  of  attractiveness). 
Each  judgement  should  reflect  a  collective  view  of  increased  overall  attractiveness,  and  therefore 
judgmental  disagreement  or  hesitation  between  two  or  more  consecutive  categories  is  allowed.  The 
judgements  were  introduced  in  the  last  column  of  the  MACBETH  weighting  judgments  shown  in  Figure 
5.  Then,  the  Tender  Commission  was  asked  to  fill  in  the  rest  of  the  matrix  (if  two  criteria  have  the  same 
weight,  “no”  is  introduce  in  the  matrix)  by  qualitatively  judging,  for  each  pair  of  criteria  j  and  j’  with  kj  > 
kj the  difference  of  overall  attractiveness  between  swinging  from  neutral  to  good  in  j  instead  of  swinging 
from  neutral  to  good  in  j’  —  that  is,  judging  the  difference  of  overall  attractiveness  between  the 
hypothetical  proposals  bj  (good  in  j  and  neutral  in  the  other  criteria)  and  by  (good  in  j’  and  neutral  in  the 
other  criteria).  Each  time  a  judgement  was  introduced  into  the  matrix,  the  software  automatically  tested  the 
consistency  of  all  the  judgments  thereto  made  and  offered  suggestions  to  resolve  inconsistencies  when 
they  arose.  For  the  consistency  matrix  of  judgments  in  Figure  4,  MACBETH  suggested  numerical  values 
for  the  weights,  which  were  subsequently  validated  and  adjusted  by  the  Tender  Commission  within  ranges 
compatible  with  the  judgments  elicited.  The  bar  chart  in  Figure  5  shows  the  relative  weighing  (in 
percentages)  finally  defined  for  the  four  evaluation  criteria. 


In  the  case  of  the  criterion  “Aspects  regarding  Technical  requirements”,  this  procedure  was  applied  firstly 
for  each  group  of  sub-sub-criteria  sharing  the  same  parent  sub-criterion,  then  for  the  group  of  seven  sub¬ 
criteria. 
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The  result  of  this  work  was  used  in  the  evaluation  regulation,  which  constituted  the  Annex  I  to  the  Tender 
Program.  This  document  described  the  Evaluation  Methodology  and  listed  the  criteria,  sub-criteria  and 
sub-sub-criteria,  as  well  as  their  relative  weighting.  For  each  relevant  area  the  regulation  also  defined  the 
characteristics  to  consider  in  the  evaluation,  as  well  as  all  of  the  reference  good  and  neutral  performances 
levels  defined. 


Figure  5  -  Using  the  m-macbeth  software  in  weighting  the  four  top-level  criteria. 

3.3  Analysis  and  Evaluation  Activities:  Scoring  the  Proposals 

After  the  reception  of  the  final  proposals  and  the  respective  formal  analysis,  the  evaluation  process  was 
initiated  with  the  verification  of  the  fulfillment  of  the  mandatory  requirements  (rejection  criteria).  Passed 
this  phase,  the  evaluation  of  the  proposals  was  conducted  taking  in  account  the  established  evaluation 
criteria.  For  this  purpose,  data  regarding  each  of  the  characteristics  defined  in  the  evaluation  regulation 
was  compiled.  The  work  progressed  in  decision  conferences  of  the  Tender  Commission,  for  assessing, 
firstly,  each  proposal  performance  in  relation  to  the  reference  performances  Good  and  Neutral  defined  in 
the  evaluation  regulation.  The  proposals  were  then  evaluated  qualitatively  regarding  their  attractiveness 
with  respect  to  each  criterional  group  of  intertwined  characteristics  -  typically  a  sub-sub-criterion. 

Figure  6  illustrates  a  graph  used  to  support  the  evaluation  of  the  proposals  on  a  “Technical  Requirements” 
sub-sub-criterion  with  ten  different  characteristics  (I  to  X).  The  graph  contains  two  broken-lines,  defining 
the  profiles  resulting  from  the  analysis  of  the  two  proposals  on  each  characteristic.  Each  vertical  line  refers 
to  one  characteristic,  with  three  zones  of  intrinsic  attractiveness:  “negative  proposal”  (if  worse  that 
Neutral),  “positive  proposal”  (if  better  than  or  indifferent  to  Neutral  and  worse  than  Good)  and  “very 
positive  proposal”  (if  better  or  indifferent  to  Good).  Note  in  Figure  6  that  the  positioning  of  a  proposal 
within  a  zone  admits  several  nuances  as,  for  instances,  the  qualification  “below  Neutral”  in  the  negative 
zone,  or  the  differentiation  between  “above  Neutral”  and  “close  to  Good”  in  the  positive  zone. 
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Figure  6  -  Example  of  a  support  graph  used  for  comparing  the  proposals  on  an  evaluation 

criterion  with  multiple  characteristics. 


Based  on  the  analyses  done  for  the  characteristics  of  a  sub-sub-criterion,  the  Tender  Commission  pairwise 
compared  the  proposals  and  the  two  reference  levels,  making  MACBETH  judgments  at  the  level  of  the 
sub-sub-criterion.  Thus,  for  each  sub-sub-criterion  (or  for  each  criterion  or  sub-criterion  with  no  children 
criterion)  a  matrix  of  qualitative  MACBETH  judgments  was  filled,  the  consistency  of  the  judgments  was 
verified  and  scores  were  assigned  to  the  proposals,  as  illustrated  in  Figure  7. 
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Figure  7  -  Example  of  a  MACBETH  evaluation  of  proposals. 


The  application  of  model  (1)  to  the  scores  of  each  proposal  on  the  sub-sub-criteria  of  each  of  the  three 
Requirements  sub-criteria,  taking  in  account  the  relative  weights  of  the  respective  sub-sub-criteria, 
resulted  in  three  aggregated  scores  for  each  proposal  on  each  one  of  those  three  sub-criteria.  Finally, 
applying  again  model  (1)  for  the  three  aggregated  scores,  taking  into  account  the  relative  weights  of  the 
three  sub-criteria,  enable  to  calculate  the  overall  score  of  each  proposal  on  the  “Technical  Requirements” 
criterion. 

A  consistent  MACBETH  matrix  of  judgments  of  difference  in  attractiveness  on  the  criterion  ‘Aspects 
regarding  Costs’  enabled  the  scoring  of  the  proposals  on  this  criterion.  In  what  concerns  the  ‘Aspects 
regarding  Offsets'  criterion  the  scores  of  the  proposals  were  calculated  by  a  linear  value  functions  defined 
upon  ‘Offsets  scores’  calculated  by  the  “Offsets  Commission”  using  a  specific  model  establish  by  law. 
Also,  value  functions  defined  for  the  13  sub-criteria  of  ‘Delivery  Schedule’  criterion  enabled  to  score  the 
options  at  this  sub-level  and  then  model  (1)  was  applied  to  calculate  proposals’  scores  on  ‘Delivery 
Schedule’.  Of  course,  the  value  functions  were  defined  in  the  Evaluation  Regulation. 
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Finally,  the  application  of  model  (1)  to  the  scores  of  each  proposal  on  the  four  top-level  evaluation  criteria, 
taking  in  account  the  respective  weights,  resulted  in  the  global  score  of  each  proposal,  as  shown  in  Figure 
8.  This  Figure  shows  that  the  difference  between  the  global  scores  of  the  two  proposals  was  about  10 
overall  value  units,  that  is,  about  10%  of  the  scoring  difference  between  a  Good  hypothetical  proposal  and 
a  Neutral  hypothetical  proposal. 
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Figure  8  -  Calculating  the  global  scores  of  the  proposals. 


3.4  Validation  and  the  Recommendation  Activities 

At  the  end,  the  results  were  validated  by  analyzing  their  sensitivity  and  robustness  to  variations  in  several 
parameters  of  the  evaluation  model  constructed.  For  instances,  it  was  analyzed  the  effect  in  the  global 
ranking  of  the  Proposals  of  changing  the  relative  weight  of  the  Requirements  criterion,  since  this  was  the 
only  criterion  where  the  Tenderer  with  the  less  attractive  proposal  presented  a  more  attractive  evaluation. 
The  conclusion  of  the  analysis  was  that  the  most  attractive  proposal  would  only  loose  its  first  place  if  the 
weighting  coefficient  of  the  factor  “Aspects  regarding  Technical  requirements”  would  become  larger  than 
57.5%  (see  Figure  9). 

All  considered,  it  was  possible  to  confirm  that  the  choice  of  best  proposal  based  on  the  judgments  issued 
by  the  Tender  Commission  and  from  the  application  of  the  evaluation  methodology  was  a  robust  decision. 
Thus,  the  Commission  wrote  the  Preliminary  Report  of  the  Negotiations  Phase  that,  after  the  ministerial 
homologation  and  a  period  of  Tenderers  Hearing,  resulted  in  the  Final  Report  of  the  Negotiations  Phase. 
With  the  homologation  of  the  Final  Report,  the  Tender  was  adjudicated  to  the  Tenderer  that  presented  the 
globally  more  attractive  proposal. 

Still  regarding  the  Evaluation  Methodology,  the  criterion  “Aspects  regarding  Costs”  included  the  cost  of  a 
contract  for  the  supply  of  spare  parts  of  the  power  train  for  all  AWV.  The  terms  and  the  conditions  for 
such  contract  were  defined,  ab  initio,  in  the  Tender  Program.  The  Portuguese  State  reserved  the  right  to 
sign  a  contract  based  on  such  proposed  conditions.  Since  the  conditions  submitted  by  the  Tenderer  who 
presented  the  best  global  proposal  were  considered  advantageous  for  the  State,  the  final  decision  was  to 
sign  a  second  Contract  for  the  Supply  of  Spare  Parts  for  the  AWV,  covering  a  10  years  period  of  the  life  of 
each  vehicle. 
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Figure  9  -  Weighting  sensitivity  analysis  on  Requirements. 


4.  CONCLUSION 

The  legislation  forces  public  tenders  for  acquisition  of  goods  and  services  to  include  an  evaluation 
regulation  that  guarantees  the  rigor  and  transparency  of  the  decision  making  process.  However,  no  specific 
methodologies  of  evaluation  are  imposed.  In  the  process  regarding  the  Program  of  Acquisition  of  8x8 
AWV  for  the  Army  and  the  Navy,  it  was  adopted  the  MACBETH  approach,  which  is  characterized  by  a 
participatory  group  decision  making  process,  based  on  the  analysis  of  the  attractiveness  of  the  proposals 
submitted  by  the  tenderers.  Thus  being,  the  application  of  this  approach  compels  the  decision  makers  to 
become  actively  involved  in  the  evaluation  of  all  the  relevant  characteristics  of  the  proposal,  through 
discussion  sessions  intended  to  the  judgment  of  the  proposals.  This  socio-technical  process  favors  the  raise 
of  awareness  about  the  implications  of  each  option.  This  approach  also  offers  methodological  consistency 
and  means  to  assess  the  robustness  of  the  results.  In  fact,  the  MACBETH  methodology  has  been  adopted, 
at  national  and  international  level,  in  several  large  Public  Tenders. 

It  is  noteworthy  that,  through  this  logistic  process,  it  was  successfully  concluded  the  award  of  a  major 
joint  program  intended  to  equip  the  Portuguese  Armed  Forces,  which  dealt  not  only  with  the  problem  of 
acquisition  of  equipment,  but  also  with  the  problem  of  their  support  over  a  significant  part  of  life  cycle. 

After  this  Tender  some  other  Defense  Tenders  have  been  conducted  or  prepared  at  the  MoD  using  the 
MACBETH  approach  to  support  the  definition  of  the  Evaluation  Regulation  and  of  the  decision-making 
process. 
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